Augmented reality (AR) technologies enable users to interact with virtual content in fundamentally new ways. AR technologies capture input from a user's surroundings, such as video data, depth sensor data, and/or audio data, and they overlay a virtual presentation directly on the user's perception of the real world using presentation devices including but not limited to smartphones, head-mounted displays (HMDs), and automotive windshields. While efforts to commercialize augmented reality technologies are relatively young, they are beginning to capture the attentions of users worldwide. For example, the wildly popular mobile augmented reality app Pokémon Go brought in over $600 million in revenue in its first three months after release, making it the most successful mobile game in history. However, the potential of augmented reality lies far beyond simple smartphone games, and we are beginning to see rapid growth in new augmented reality technologies. For example, the HoloLens HMD is now available from Microsoft Corporation to developers internationally, Meta's second-generation HMD is available for pre-order, and Google has invested over $500 million in the HMD startup Magic Leap. Additionally, many groups within the automotive industry are developing AR-enabled windshields to aid drivers. Overall, interest in employing AR technologies across diverse industry sectors is increasing, with AR as a whole projected to grow into a $100 billion industry by the year 2020.
Though AR technologies have the potential to deliver tremendous benefits, they also raise new privacy and security risks. A growing body of literature focuses on mitigating privacy risks that stem from applications' needs to gather input from the numerous sensors on AR devices, such as cameras. This literature does not, however, address the complementary issue of the risks that arise from AR applications' abilities to modify the user's view of the world, which constitute security risks of AR output. Addressing these risks is particularly critical for fully immersive AR systems, such as HMDs and car windshields, where users cannot easily disengage from their devices if output security issues arise.
To illustrate potential security risks related to AR output, imagine driving a car with an AR-enabled windshield. Some potential features of this technology include the ability to visibly highlight lane markers to prevent accidental lane drift, to display turn-by-turn driving directions visually overlaid on the road, and to visibly warn the driver of impending collisions. These tasks might run on an AR system as multiple components of a single application, or as multiple, distinct applications. Without appropriate output safeguards, however, the benefits of these applications can be overshadowed by risks. For example, a malicious or buggy AR application could potentially obscure real-world pedestrians, overlay misleading information on real-world road signs, or occlude the virtual content of other AR applications, such as collision warnings or other important safety alerts. Similar issues could arise with HMDs for a user on foot. Consider, for example, an HMD application that accidentally or intentionally blocks the user's view of a tripping hazard or an oncoming car.
To our knowledge, no existing industry or research AR platforms are designed to mitigate the above types of output security risks. Today, it is the responsibility of the applications themselves to safely generate output, and AR systems do not themselves enforce any types of output policies. Placing this responsibility with application developers who may generate buggy, vulnerable, or malicious code, is not reliable. Furthermore, the fact that today's AR platforms cannot exert any control over the output from individual applications means they also cannot handle conflicts between the output from multiple applications. HoloLens sidesteps this problem by simply not supporting multiple full-screen immersive AR applications running at once, and so is not even capable of providing this functionality.
What is needed are improvements to augmented reality systems to provide output security, even in the face of multiple concurrently executing untrusted applications. It is also desirable to provide output security without unduly impacting overall system performance.